Learning Shaping Rewards in Model-based Reinforcement Learning

Potential-based reward shaping has been shown to be a powerful method to improve the convergence rate of reinforcement learning agents. It is a flexible technique to incorporate background knowledge into temporal-difference learning in a principled way. However, the question remains how to compute the potential which is used to shape the reward that is given to the learning agent. In this paper, we show how, in the absence of knowledge to define potential manually, the potential function can be learned online in parallel with the actual reinforcement learning process. The approach for the prototypical model-based R-max algorithm is proposed. It learns the potential function using the free space assumption about the transitions in the environment. The novel algorithm is presented and evaluated empirically and theoretically. Specifically, the proposed algorithm is shown to learn an admissible potential which is required by the R-max algorithm with potential-based reward shaping.